Quad-Byte Transformation using Zero-frequency Bytes

Byte pair encoding (BPE) algorithm was suggested by P. Gage is to achieve data compression. It encodes all instances of most frequent byte-pair using zerofrequency byte in the source data. This process is repeated for maximum m possible number of passes until no further compression is possible, either because there are no more frequently occurring byte pairs or there are no more unused zero-frequency bytes to represent pairs. The proposed quadbyte transformation algorithm encodes 4-byte integers instead of byte pairs. Proposed QBT-Z (Quad-Byte Transformation using Zero-frequency bytes) is applied on data blocks of a file. Implementation of block-wise QBT-Z with varying number of passes is experimented with various test files. The aim is to minimize the compression time and achieve better compression rate. QBT-Z with m passes transforms only one quad-byte at a time using unused zero-frequency byte and the process is repeated maximum m possible number of times. With k-pass QBT-Z, k<<m, it transforms half of the possible most-frequent quad bytes in each pass except the last. In the last pass, it transforms all remaining possible quad-bytes. Algorithm is tested on 18 files of various types with a total size of nearly 40MB. Overall transformation rate (compression rate using transformation) is 17.04%, 18.91%, 19.25%, 19.31% and 22.75% using QBT-Z with 1, 2, 3, 4 and m passes respectively. It shows that transformation with large number passes increases reduction in transformed file size. Total transformation time taken by k-pass QBT-Z with 1, 2, 3 and 4 passes is only 7.629, 12.183, 16.091 and 20.994 seconds respectively; which is significantly small as compared to 268.434 seconds taken by m-pass QBT-Z. As the number of passes increases, compression is better with increased execution time. Reverse transformation is too fast, hardly taking 2 to 4 seconds. Keywords—data compression, transformation, quad-byte encoding, substitution using zero-frequency bytes